This script is the visual inspection scripts. 1. It takes the data file on EDI and combines it with the L1 file which has already been QAQCed. 2. Then the script checks for daily and subdaily gaps in the current file. 3. Lists the flag frequency to check if there are any NAs or any assigned the wrong flag. 4. If necessary can QAQC data already on EDI using the QAQC function 5. Creates plots 6. Writes data to new csv

For the plots, they use a function called “all_plot”. In all_plot you can specify if you want plotly plots for the current data. BEWARE if you turn on the plotly plots and try to knit the markdown it will fail! I am working on a fix. For right now you can specify which plotly plots you want on. You can also look at the plotly plots manually in each chunk by running the chunk with Use_plotly=TRUE as an argument and then at the end of the chunk output[[1]].

All files are from the link to GitHub and the source scripts are from GitHub as well.

Check for gaps in the data frame

This identifies if there are any daily data gaps in the long-term record

## [1] "2023-08-08 12:20:00 UTC" "2023-08-14 11:30:00 UTC"
## [1] "2023-08-19 04:20:00 UTC" "2023-08-21 09:30:00 UTC"

This identifies if there are any sub-daily gaps in the current record, as gaps of the long term record are found in the metadata.

The first row is the time for the first observation and then the subsequent observation. Each observation should be 10 minutes apart. The second row is the number of the record for each observation. Most of these gaps happen when we change the program on the data logger. These times will be recorded in the maintenance section of the metadata and are also noted in the maintenance log.

## [1] "2023-03-13 14:00:00 UTC" "2023-03-13 14:20:00 UTC"
## [1] 1882    0
## [1] "2023-06-07 12:00:00 UTC" "2023-06-07 12:30:00 UTC"
## [1] 2 0
## [1] "2023-06-27 11:00:00 UTC" "2023-06-27 11:20:00 UTC"
## [1] 2871    0
## [1] "2023-08-08 12:20:00 UTC" "2023-08-14 11:30:00 UTC"
## [1] 6054 6913
## [1] "2023-08-19 04:20:00 UTC" "2023-08-21 09:30:00 UTC"
## [1] 7590 7909

QAQC Historical Files: If you need to QAQC the historical file then run it through the qaqc function without start_date and end_date as NULL

Read in the QAQC File and Check it out

This section reads in the QAQC file and then you can look at the head, tail and structure. Make sure the last row is Dec. 31 23:50 of the publishing year. There should be 85 columns unless a new one has been added.

Information for EDI methods text

This is information that is included in the methods file for EDI

The minimum water level for BVR was 15.6535017 and the maximum water depth was 19.6358085.

The median water level was 18.5887838 and the mean water level was 18.4561891.

Flag Frequency

Let’s look at the flag Frequency for each variable. As a reminder here are the flag codes Flag values

0: no flag

1: value removed due to maintenance and set to NA

2: negative or outlier value removed and set to NA, see Methods section for more detail on QAQC process

3: negative values set to 0 except for temperature

4: value removed due to fouling and set to NA

5: questionable value but left in the dataset

6: Values adjusted using a linear or square root function to match high-resolution CTD profiles and corrected other affected observations on the same sensor

7: missing data

## Flag_ThermistorTemp_C_1
##     0     2     7  <NA> 
## 43664 93965  4633     0 
## Flag_ThermistorTemp_C_2
##      0      2      7   <NA> 
## 104015  33616   4631      0 
## Flag_ThermistorTemp_C_3
##      0      2      7   <NA> 
## 123158  14473   4631      0 
## Flag_ThermistorTemp_C_4
##      0      2      7   <NA> 
## 136597   1034   4631      0 
## Flag_ThermistorTemp_C_5
##      0      2      7   <NA> 
## 137582     48   4632      0 
## Flag_ThermistorTemp_C_6
##      0      2      7   <NA> 
## 137582     48   4632      0 
## Flag_ThermistorTemp_C_7
##      0      2      7   <NA> 
## 137582     48   4632      0 
## Flag_ThermistorTemp_C_8
##      0      2      7   <NA> 
## 137579     48   4635      0 
## Flag_ThermistorTemp_C_9
##      0      2      7   <NA> 
## 137582     48   4632      0 
## Flag_ThermistorTemp_C_10
##      0      2      7   <NA> 
## 137578     48   4636      0 
## Flag_ThermistorTemp_C_11
##      0      2      7   <NA> 
## 137583     48   4631      0 
## Flag_ThermistorTemp_C_12
##      0      2      7   <NA> 
## 137582     48   4632      0 
## Flag_ThermistorTemp_C_13
##      0      1      2      7   <NA> 
## 134740   2843     48   4631      0 
## Flag_EXOTemp_C_1
##      0      1      2      7   <NA> 
## 105029     51   5163  32019      0 
## Flag_EXOCond_uScm_1
##      0      1      2      7   <NA> 
## 105029     51   5163  32019      0 
## Flag_EXOSpCond_uScm_1
##      0      1      2      7   <NA> 
## 105028     51   5164  32019      0 
## Flag_EXOTDS_mgL_1
##      0      1      2      7   <NA> 
## 105028     51   5164  32019      0 
## Flag_EXODOsat_percent_1
##      0      1      2      7   <NA> 
## 104804     51   5163  32244      0 
## Flag_EXODO_mgL_1
##      0      1      2      7   <NA> 
## 104804     51   5163  32244      0 
## Flag_EXOChla_RFU_1
##      0      1      2      3      7   <NA> 
## 104880     51   5301     11  32019      0 
## Flag_EXOChla_ugL_1
##     0     1     2     3     6     7  <NA> 
## 71833    51  5286  1449 31624 32019     0 
## Flag_EXOBGAPC_RFU_1
##     0     1     2     3     7  <NA> 
## 76503    51  5170 28519 32019     0 
## Flag_EXOBGAPC_ugL_1
##     0     1     2     3     6     7  <NA> 
## 31000    51  5172 56150 17870 32019     0 
## Flag_EXOfDOM_RFU_1
##      0      1      2      7   <NA> 
## 105011     51   5181  32019      0 
## Flag_EXOfDOM_QSU_1
##      0      1      2      7   <NA> 
## 105011     51   5181  32019      0 
## Flag_EXOPressure_psi_1
##      0      1      2      7   <NA> 
## 104986     51   5206  32019      0 
## Flag_EXODepth_m_1
##      0      1      2      7   <NA> 
## 104985     51   5207  32019      0 
## Flag_EXOBattery_V_1
##      0      1      2      7   <NA> 
## 105027     51   5163  32021      0 
## Flag_EXOCablepower_V_1
##      0      1      2      7   <NA> 
## 105029     51   5163  32019      0 
## Flag_EXOWiper_V_1
##     0     1     2     7  <NA> 
## 71916 16341  5163 48842     0 
## Flag_EXOTemp_C_9
##      0      1      2      7   <NA> 
## 110501    619   4031  27111      0 
## Flag_EXOCond_uScm_9
##      0      1      2      7   <NA> 
## 110308    619   4224  27111      0 
## Flag_EXOSpCond_uScm_9
##      0      1      2      7   <NA> 
## 110308    619   4224  27111      0 
## Flag_EXOTDS_mgL_9
##      0      1      2      7   <NA> 
## 110308    619   4224  27111      0 
## Flag_EXODOsat_percent_9
##      0      1      2      7   <NA> 
## 110294    619   4033  27316      0 
## Flag_EXODO_mgL_9
##      0      1      2      7   <NA> 
## 110296    619   4031  27316      0 
## Flag_EXOfDOM_RFU_9
##      0      1      2      7   <NA> 
## 109391    619   5141  27111      0 
## Flag_EXOfDOM_QSU_9
##      0      1      2      7   <NA> 
## 109391    619   5141  27111      0 
## Flag_EXOPressure_psi_9
##      0      1      2      7   <NA> 
## 110501    619   4031  27111      0 
## Flag_EXODepth_m_9
##      0      1      2      7   <NA> 
## 110501    619   4031  27111      0 
## Flag_EXOBattery_V_9
##      0      1      2      7   <NA> 
## 110501    619   4031  27111      0 
## Flag_EXOCablepower_V_9
##      0      1      2      7   <NA> 
## 110501    619   4031  27111      0 
## Flag_EXOWiper_V_9
##      0      1      2      7   <NA> 
## 105936   5184   4031  27111      0 
## Flag_LvlPressure_psi_13
##      0      2      7   <NA> 
## 137583     48   4631      0 
## Flag_LvlTemp_C_13
##      0      2      7   <NA> 
## 137584     47   4631      0

QAQC Plots

For the plots, they use a function called “all_plot”. In all_plot you can specify if you want plotly plots for the current data. BEWARE if you turn on the plotly plots and try to knit the markdown it will fail! I am working on a fix. For right now you can specify which plotly plots you want on. You can also look at the plotly plots manually in each chunk by running the chunk with Use_plotly=TRUE as an argument and then at the end of the chunk output[[1]].

Temperature

## Warning: Removed 32252 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 98598 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 679 rows containing non-finite values (`stat_density()`).

## Warning: Removed 679 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 21184 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 38247 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 262 rows containing non-finite values (`stat_density()`).

## Warning: Removed 262 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 14473 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 19104 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 131 rows containing non-finite values (`stat_density()`).

## Warning: Removed 131 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 1034 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 5665 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 38 rows containing non-finite values (`stat_density()`).

## Warning: Removed 38 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 49 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4683 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4684 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4679 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4679 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37233 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4679 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

Depth

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37277 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 4679 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32 rows containing non-finite values (`stat_density()`).

## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).

Dissolved Oxygen

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37458 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Removed 16402 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37458 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31968 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31966 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

Chlorophyll and Phycocanin

## Warning: Removed 7581 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16489 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37371 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 37371 rows containing missing values (`geom_point()`).
## Warning: Removed 83 rows containing missing values (`geom_line()`).

fDOM

## Warning: Removed 7578 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16411 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37251 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 7578 rows containing missing values (`geom_scattermore()`).
## Removed 16411 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37251 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 11715 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32871 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 11715 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 32871 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

Conductivity, Specific Conductivity, TDS

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37233 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16403 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37234 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Removed 16403 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37234 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10804 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31954 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10804 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31954 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10804 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31954 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

Wiper, Battery, and CablePower

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16404 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37235 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 37233 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 245 rows containing non-finite values (`stat_density()`).

## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Removed 16402 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 54056 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 361 rows containing non-finite values (`stat_density()`).

## Warning: Removed 361 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).

## Warning: Removed 212 rows containing non-finite values (`stat_density()`).

## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).

Write to CSV

Download and save Maintenance Log, Plotting function, and QAQC function